Repository navigation
Gateway waves 1–7: integrate the gap-analysis features - #14
Conversation
Compares MultiLLM against the 1,185 repositories tagged llm-gateway on GitHub, ranks the 40 largest gaps with evidence on both sides, and specifies the first six features to build.
Provider circuits now classify each transport result into independent provider-health, credential-health and replay signals. Credential rejection, throttling and caller errors no longer heal a degraded or open circuit, and they release half-open probes without an outage penalty. Retry rules and raw single-forward behaviour are unchanged.
Adds MULTILLM_STREAM_PREFLIGHT (off by default). In strict mode, streaming auto-route Chat requests are held until the first useful text, reasoning or tool-argument delta arrives within a bounded prefix, then the buffered bytes are replayed once in order. Empty, malformed or errored prefixes return a real 502 or 504 and never trigger another upstream request.
Adds scripts/provider_authenticity.py, an operator tool that checks one provider-qualified model through the gateway. By default it sends one metadata request and no generation. With --allow-generation it runs at most two 64-token nonce probes and reports narrow supported, contradicted or unknown verdicts without raw prompts, completions or credentials.
Adds ROLEPLAY_CANDIDATE_CONTEXT_REFIT (off by default). In window mode each fallback candidate gets a context view that fits its own window: the oldest complete unprotected message groups are omitted while directives, the latest user turn and tool-call pairs stay intact. Payloads, prompt-cache annotations, continuation and headers use the selected view; stored history and recovery keep the full conversation.
…nto integrate/gateway-wave2
…egrate/gateway-wave2
…grate/gateway-wave2
… integrate/gateway-wave2
… into integrate/gateway-wave2
The Worker now passes MULTILLM_STREAM_PREFLIGHT through its Container environment allowlist, so the setting works on the Cloudflare deployment as well as standalone Flask. The README lists the feature and the guide describes deployment.
The guide now states that the classification needs no configuration.
The guide now describes how the candidate context tests run.
The README feature list links the diagnostics guide.
POST /v1/messages/count_tokens accepts X-MultiLLM-Token-Count-Mode. estimate stays the default and keeps its contract. auto uses a configured provider count and falls back to a labeled estimate. native requires a provider count and returns 501 when none is configured.
NATIVE_TOKEN_COUNT_ENDPOINTS_JSON maps a registered provider to its verified count path and defaults to {}. The adapter sends one bounded, count-only request with redirects and retries disabled. Upstream errors keep their status in an Anthropic error envelope. Only native counts reach a provider, so only they check the key's model allowlist.
…egrate/gateway-wave2
…grate/gateway-wave2
… integrate/gateway-wave2
… into integrate/gateway-wave2
…tegrate/gateway-wave2
PROMETHEUS_ENABLED=true serves two Prometheus 0.0.4 text endpoints and defaults to off, where both return 404. /status.prometheus is public. The Worker renders it from the D1 status snapshot, so a scrape never wakes the Container, and it exposes only the fixed public route, provider and overall states. /v1/metrics/prometheus requires an admin user with an admin-scoped API key. Flask renders it with aggregate 24-hour request counts by response class plus p50/p95 latency. Both endpoints cap output at 512 series and 256 KiB and report truncation in a gauge.
…egrate/gateway-wave2 # Conflicts: # package.json # tests/fixtures/secret_firewall_coverage.json
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configuration
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e0c6b6644f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| state, | ||
| candidate, | ||
| messages: roleplayMessages, | ||
| messages: selectedMessages, |
There was a problem hiding this comment.
Preserve per-candidate windows during refusal fallback
When a large-window GLM candidate returns the semantic-refusal sentinel and the configured uncensored fallback has a smaller context window, passing only selectedMessages here loses the fallback candidate's previously computed contextPlan. createRoleplayContinuation recalculates capacity from the first candidate's larger view without refitting and can report roleplay_continuation_context_exhausted, returning the generic fallback-failed response without ever dispatching the otherwise eligible smaller candidate. Carry the per-candidate view into semantic-refusal dispatch rather than reusing the selected candidate's messages for every fallback.
Useful? React with 👍 / 👎.
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| 🔵 In progress View logs |
multillm-proxy | f33ffa7 | Oct 09 2026, 10:41 AM |
Provider catalogs can declare a model's precision (fp32, fp16, bf16, fp8, int8, int4) and quantization scheme. MODEL_PRECISION_PREFERENCE takes an ordered JSON list of precisions and is unset by default. While unset, catalog output and auto:intelligence candidate order are unchanged. When set, precision is recorded only from a provider's explicit catalog fields, never inferred from a model name. It breaks ties between candidates the quality and fast profiles already rank equally. It never adds a candidate, crosses a ranking group, reorders the balanced chain or changes an explicitly requested model. The intelligence policy can override the setting with its own precision_preference list. An invalid environment value is logged once and treated as unset.
RESPONSE_CACHE_POLICY_REVISION selects the response cache identity. legacy, the default (also when unset or empty), keeps today's keys, bodies and headers. v1 adds a canonical digest of the resolved route, the key's model permissions, its secret-scan mode, the retention policy and the workflow version. A policy change therefore misses instead of replaying an answer produced under different rules. Under v1, routes whose effective plan is not fully known (free pools, cascades, health-ordered auto routes, query parameters or provider-selection headers) bypass the cache. An answer is not stored if the policy changed while it was being generated. Unknown revisions bypass the cache, and old entries are never purged, only left unreachable.
scripts/probe_cost_forecast.py reads a JSON plan (from a file or stdin) describing the probe schedule, routes and prices. It prints upper bounds on provider calls and USD per day, per month and per run, and exits nonzero when a supplied --max-calls or --max-cost-usd limit would be exceeded. The script never runs probes or contacts providers, and does not read the application environment. A price the plan does not state is reported as null, never as zero.
GET /openapi.json returns an OpenAPI 3.1 description of the public client API: Chat Completions, Responses, Anthropic Messages and token counting, models and image generation, including the opencode and codex-easy paths. It also covers the documented MultiLLM headers and the error and SSE envelopes. It requires a dashboard login or an API key with the models scope. services/openapi_spec.py builds the spec without reading provider configuration, and docs/openapi.json is its committed output. A test fails when the two drift. Administrator routes are excluded.
Three tests started this machine's virtualenv Python by absolute path, so they failed on GitHub Actions once the dependency audit stopped masking them. The Worker tests now use PYTHON, then .venv, then python3, as the D1 integration test does, and the node job installs the locked Python dependencies.
On a slow runner the 20 ms deadline expired before the request reached the fetcher, so the hold was released instead of kept as unknown. The fetcher never settles, so a 1000 ms deadline still fires after handoff.
The test checks admission uses the 1000 ms routing deadline rather than the 5000 ms policy one; a 3000 ms bound still tells them apart.
… test The request takes about 80 ms, but a loaded GitHub runner took 530 ms. The call count assertion already rules out a second attempt.
…ails An intermittent CI failure returned 502 with no log, so the assertion now reports the provider call count and the response body.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
feat(gateway): wire the Wave 7 enterprise features together
feat(gateway): integrate the Wave 6 gateway features
feat(gateway): integrate the Wave 5 gateway features
feat(gateway): integrate the Wave 4 gateway features
Integrate Wave 3 gateway features
Integrate Wave 2 gateway features
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
|---|---|---|---|---|---|
| 37552414 | Triggered | Generic Password | c8787fc | worker/alert-delivery.mjs | View secret |
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secret safely. Learn here the best practices.
- Revoke and rotate this secret.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
Now carries Waves 1–7
The integration stack (#25, #39, #53, #60, #69, #76) was merged into this branch from the top down, so merging this PR lands all seven waves (#7–#76) and the dependency-audit fix (#42) on main. This branch's tree is identical to
integrate/gateway-waves-1-7at27c5f1d, which passed every CI job.D1 migrations
0014through0038were applied tomultillm-intelligencebefore this merge. Every new feature is off by default. The sections below describe Wave 1 as it was first opened.Combines Wave 1 of the gateway gap analysis into one branch, so the features can be reviewed and merged together.
Included
auto:Chat streamsMULTILLM_STREAM_PREFLIGHT=offROLEPLAY_CANDIDATE_CONTEXT_REFIT=off/v1/messages/count_tokensestimatemode,NATIVE_TOKEN_COUNT_ENDPOINTS_JSON={}PROMETHEUS_ENABLED=falseMerging this PR merges all seven, and closes them as merged once GitHub sees their commits on main.
Conflict resolution
package.jsontest:worker: bothtests/test_roleplay_candidate_context_worker.mjs(feat(roleplay): fit fallback context to each candidate window #11) andtests/test_prometheus_worker.mjs(feat(metrics): add opt-in Prometheus exposition #13) are registered.tests/fixtures/secret_firewall_coverage.json: bothworker/roleplay/candidate-context.mjsandworker/prometheus.mjsare inedge_modules, in sorted order.worker/container-env.mjskeys merged without conflicts.Verification on the merged tree (clean worktree, no
.env)test_proxy_documentation: 21 passed.npm run test:worker: 1204/1204.test:deploy13/13, free routes 8/8, client headers 12/12.cf:migrations:checkpasses; no new migrations.wrangler deploy --dry-runbundles: 786.64 KiB, 174.45 KiB gzip.tests/test_frontend_build.mjsfails locally on main too, because@tailwindcss/cliisn't installed here.Deploy note
Merging to main deploys through Workers Builds. With the defaults above, live behaviour is unchanged apart from #8's outcome classification. No D1 migration is needed.